{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seting up AWS S3 bucket to run TAO Toolkit training process \n",
    "* Install necessary os packages \n",
    "* Since this notebook requires a running several commands as `sudo`, please enable sudoers with NOPASSWD.\n",
    "  You may do so by using the command below:\n",
    "  \n",
    "  ```sh\n",
    "      ubuntu ALL=(ALL) NOPASSWD: ALL\n",
    "  ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install necessary apt packages to mount the s3 storage as a unix folder\n",
    "!sudo apt update\n",
    "!sudo apt-get upgrade -y\n",
    "!sudo apt-get -y install build-essential git libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support automake libtool\n",
    "!sudo apt-get -y install pkg-config libssl-dev\n",
    "!sudo apt install s3fs -y\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create password cridentials to set access to the S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the AWS S3 access key\n",
    "import os\n",
    "os.environ[\"S3FS_PWD\"] = FIXME\n",
    "!sudo sh -c \"echo $S3FS_PWD > /root/.passwd-s3fs\"\n",
    "!sudo chmod 600 /root/.passwd-s3fs\n",
    "!sudo ls -l /root/.passwd-s3fs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attach the S3 bucket to the VM as a unix mountpoint with s3fs\n",
    "\n",
    "* the script over writes the fstab entries.\n",
    "* if a previous mount is already present the step will report failure.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mount the S3 storage to the runtime environment as a unix folder\n",
    "import os\n",
    "os.environ[\"S3_DATA_MOUNT\"]=\"/mnt/data\"\n",
    "os.environ[\"S3_URL\"]=FIXME\n",
    "os.environ[\"MOUNTPOINT\"]=\"tao-test-data-01\"\n",
    "os.environ[\"HOME\"]=\"root\"\n",
    "!if [ ! -d /tmp/cache ]; then sudo mkdir /tmp/cache; else echo \"/tmp/cache already exists\"; fi ;\n",
    "!sudo chmod -R 777 /tmp/cache\n",
    "!if [ ! -d $S3_DATA_MOUNT ]; then  sudo mkdir -p $S3_DATA_MOUNT;  else echo \"$S3_DATA_MOUNT already exists\"; fi\n",
    "!sudo s3fs -o use_cache=/tmp/cache -o allow_other -o umask=0022 -o url=\"$S3_URL\" $MOUNTPOINT $S3_DATA_MOUNT\n",
    "!FIXME\n",
    "#You may also mount by updating the mount path to in the /etc/fstab file\n",
    "#The following commands also back up the previous fstab file.\n",
    "!sudo cp /etc/fstab /etc/fstab-`date +%d-%m-%Y-%s`; \\\n",
    " sudo sh -c \"echo s3fs#$MOUNTPOINT $S3_DATA_MOUNT fuse _netdev,allow_other 0 0 >> /etc/fstab\"; \\\n",
    " sudo mount $S3_DATA_MOUNT\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copy test data to a local storage\n",
    "After the data is copied to the local storage you may launch other notebooks to continue the training and other functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy test data to a local storage directly attached to the system\n",
    "# Running training with dataset mounted on cloud storages \n",
    "# Will slow the training process due to network bandwidth limitations.\n",
    "\n",
    "# Make sure you have enough storage available under the LOCAL_PROJECT_DIR\n",
    "import os\n",
    "os.environ[\"LOCAL_PROJECT_DIR\"]=FIXME # this is the path to where the Sample notebooks are downloaded.\n",
    "!cp -rp $S3_DATA_MOUNT $LOCAL_PROJECT_DIR; \\\n",
    "ls $LOCAL_PROJECT_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip3 install awscli\n",
    "\n",
    "# os.environ[\"DRIVE_NAME\"]=FIXME\n",
    "# os.environ[\"AWS_ACCESS_KEY_ID\"]=FIXME\n",
    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"]=FIXME\n",
    "# !aws s3 cp <fixme> s3://${DRIVE_NAME}\n",
    "# !aws s3 ls s3://${DRIVE_NAME}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
